Python error: [Errno 24] Too many open files analysis and resolution

background

Recently, I found an error in my work, which frequently occurs when executing multithreaded scan scripts

HTTPConnectionPool(host= ' t.tips', port=80): Max retries exceeded with url: /index.php (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f29d4081150>: Failed to establish a new connection: [Errno 24] Too many open files',))

Weird...

Analysis of the

Blogger own hand 1 small projects often require multi-threaded batch scanning to collect data, such as collect data function is of course the thread, the more the better, so I normally 3000 threads are used for scanning, in determining the bandwidth is enough, the system hardware configuration under the condition of high enough will still appear afore-mentioned problems, this let me think of is because of the limitation of system problems.

Through the search engine I found an introduction to system limitations, implementation

$ ulimit -n
1024

The result is 1024. The system is limited to opening 1024 files at the same time, which is obviously too few.

To solve

The solution to this problem is simple, simply modify the following files

sudo vim /etc/security/limits.conf

Add two lines of code at the end of the file

* soft nofile 10240
* hard nofile 10240

Of course, this number can be modified according to their own needs, after saving log out and log in again.

conclusion

The above is the whole content of this article, I hope the content of this article to your study or work can bring 1 definite help, if you have questions you can leave a message to communicate.